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1 Introduction 

Let A; be a field, let G be a reductive algebraic group over k, and let V he a linear representation 
of G. Geometric invariant theory involves the study of the /c-algebra of G-invariant polynomials on V, 
and the relation between these invariants and the G-orbits on V, usually under the hypothesis that 
the base field k is algebraically closed. In favorable cases, one can determine the geometric quotient 
V//G = Spec(Sym*(y'^))'^ and can identify certain fibers of the morphism V — )■ V//G with certain 
G-orbits on V. 

As an example, consider the three-dimensional adjoint representation of G = SL2 given by 
conjugation on the space \/ of 2 x 2 matrices v = {c -a) of trace zero. This is irreducible when 
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the characteristic of k is not equal to 2, which we assume here. It has the quadratic invariant q{v) = 
— det(t') = bc + a^, which generates the full ring of polynomial invariants. Hence V//G is isomorphic 
to the affine line and q : V ^ ^//G = Ga- If v and w are two vectors in V with q{v) = q(w) ^ 0, then 
they lie in the same G-orbit provided that the field k is separably closed. 

For general fields the situation is more complicated. In our example, let d be a non-zero element 
of k and let K be the etale quadratic algebra k[x]/ {x"^ — d). Then the G{k)-oxhii% on the set of vectors 
V with q{v) = d ^ can be identified with elements in the 2-group k* /NK*. (See §2.) 

The additional complexity in the orbit picture, when k is not separably closed, is what we refer 
to as arithmetic invariant theory. It can be reformulated using non-abelian Galois cohomology, but that 
does not give a complete resolution of the problem. Indeed, when the stabilizer Gy of v is smooth, we 
will see that there is a bijection between the different orbits over k which lie in the orbit of v over the 
separable closure and the elements in the kernel of the map in Galois cohomology 7 : H^{k, Gy) — )■ 
H^{k, G). Since 7 is only a map of pointed sets, the computation of this kernel can be non-trivial. 

In this paper, we will illustrate some of the issues which remain by considering the regular 
semi-simple orbits — i.e., the closed orbits whose stabilizers have minimal dimension — in three rep- 
resentations of the split odd special orthogonal group G = S02n+i = SO(iy) over a field k whose 
characteristic is not equal to 2. Namely, we will study: 

• the standard representation V = W; 

• the adjoint representation V = so{W) = A'^{W); and 

• the symmetric square representation V = Sym^{W). 

In the first case, the map 7 is an injection and the arithmetic invariant theory is completely determined 
by the geometric invariant theory. In the second case, the stabilizer is a maximal torus and the arithmetic 
invariant theory is the Lie algebra version of stable conjugacy classes of regular semi-simple elements. 
The theory of stable conjugacy classes, introduced by Langlands [13]-[14] and developed further by 
Shelstad [23] and Kottwitz [10], forms one of the key tools in the study of endoscopy and the trace 
formula. Here there are the analogous problems, involving the Galois cohomology of tori, for the 
adjoint representations of general reductive groups. In the third case, there are stable orbits in the sense 
of Mumford's geometric invariant theory [17], i.e., closed orbits whose stabilizers are finite. Such 
representations arise more generally in Vinberg's invariant theory (cf. [20], [18]), where the torsion 
automorphism corresponds to a regular elliptic class in the extended Weyl group. In this case, we can 
use the geometry of pencils of quadrics to describe an interesting subgroup of classes in the kernel of 7. 

Although we have focused here primarily on the case of orbits over a general field, a complete 
arithmetic invariant theory would also consider the orbits of a reductive group over more general rings 
such as the integers. We end with some remarks on integral orbits for the three representations we have 
discussed. 

We would like to thank Brian Conrad, for his help with etale and flat cohomology, and Mark 
Reeder and Jiu-Kang Yu for introducing us to Vinberg's theory. We would also like to thank Bill 
Casselman, Wei Ho, Alison Miller, Jean-Pierre Serre, and the anonymous referee for a number of 
very useful comments on an earlier draft of this paper. It is a pleasure to dedicate this paper to Nolan 
Wallach, who introduced one of us (BHG) to the beauties of invariant theory. 
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2 Galois cohomology 



Let k he a field, let be a separable closure of k, and let k'^ denote an algebraic closure containing 
fc*. Let r be the (profinite) Galois group of A;* over k. Let G be a reductive group over k and V an 
algebraic representation of G on a finite-dimensional A;-vector space. The problem of classifying the 
G(A;)-orbits on V{k) which lie in a fixed G{k^)-orbit can be translated (following Serre [21, §1.5]) into 
the language of Galois cohomology. 

Let V E V{k) be a fixed vector in this orbit, and let be the stabilizer of v. We assume that Gy 
is a smooth algebraic group over k. If w E V{k) is another vector in the same G(/c'^)-orbit as v, then 
we may write w = g{v) with g E G{k^) well-defined up to right multiplication by For every 

(7 G r, we have g'^ = ga^ with a^r E G^{k^). The map a ^ a„ h a. continuous 1-cocycle on V with 
values in Gi,{k'^), whose class in the first cohomology set H^{T, Gy{k^)) is independent of the choice 
of g. Since = g~^g'^ , this class is trivial when mapped to the cohomology set H^(V, G{k'^)). We 
will use the notation H^{k, G„) and H^{k, G) to denote these Galois cohomology sets in this paper. 

Reversing the argument, one can show similarly that an element in the kernel of the map of 
pointed sets H^{k, G^) H^{k, G) gives rise to a G(A;)-orbit on V{k) in the G{k^)-orbit of v. Hence 
we obtain the following. 

Proposition 1 There is a bijection between the set of G{k)-orbits on the vectors w in V{k) that lie in 
the same G{k^)-orbit as v and the kernel of the map 

'y:H\k,G,)^H\k,G) (1) 

in Galois cohomology. 

When the stabilizer Gy is smooth over k, the set of all vectors w E V{k) lying in the same G{k'^)- 
orbit as v can be identified with the A;-points of the quotient variety G /G^, and the central problem of 
arithmetic invariant theory in this case is to understand the kernel of the map 7 in Galois cohomology. 
This is particularly interesting when is a finite, local, or global field, when the cohomology of the 
two groups Gy and G can frequently be computed. 

In the example of the introduction with G = SL2 and V the adjoint representation (again assum- 
ing char(A;) 7^ 2), let v be a vector in V{k) with q(v) = d ^ 0. Then the stabilizer Gy is a maximal 
torus in SL2 which is split by the etale quadratic algebra K. The pointed set H^{k, G) = H^{k, SL2) 
is trivial, so all classes in the abelian group H^{k, Gy) = k* /NK* lie in the kernel of 7. These classes 
index the orbits of SL2(A;) on the set S of non-zero vectors w with q(w) = q{v), since this is precisely 
the set S of vectors w E V{k) which lie in the same SL2(A;'^)-orbit as v. (This illustrates the point 
that one first has to solve the orbit problem over the separable closure k'^, before using Proposition 1 to 
descend to orbits over k.) 

The vanishing of H^{k,G) occurs whenever G = GL„ or G = SL„ or G = Sp2„, and gives 
an elegant solution to many orbit problems. For example, when the characteristic of k is not equal 
to 2, the classification of the non-degenerate orbits of SL„ = SL(W^) on the symmetric square repre- 
sentation V = Sym^(iy^) shows that the isomorphism classes of non-degenerate orthogonal spaces 
W of dimension n over k with a fixed determinant in k*/k*'^ correspond bijectively to classes in 
H\k,Gy) = H\k,SO{W)) (cf. [11, Ch VII, §29], [21, Ch III, Appendix 2, §4]). In general, both 
H^{k, Gy) and H^{k, G) are non-trivial, and the determination of the kernel of 7 remains a challenging 
problem. 
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Remark 2 In those cases where the stabilizer Gy is not smooth, it is at least flat of finite type over k, 
so one can replace the map 7 in Galois (etale) cohomology with one in flat (fppf) cohomology. Indeed, 
the A;-valued points of G/Gy can always be identified with the possibly larger set S' of vectors w' in 
V{k) which lie in the same G(A;")-orbit as v, where A;" is an algebraic closure of k'^. As an example, 
the group Gm acts onV = Ga by the formula X{v) = \^ ■ v. The stabilizer of f = 1 is the subgroup 
/ip, and G/Gy = Gm/ l^p = G„i- The stabilizer Gy is smooth if the characteristic of k is not equal to p, 
in which case the set S consists of the non-zero elements of the field k, and the G'(A;)-orbits on S form 
a principal homogeneous space for the group H^{k, fip) = k* / k*^. If the characteristic of k is equal to 
p, the stabilizer /ip is not smooth over k. In this case the set S consists of the p^^ powers in k*. The set 
S' is equal to the full group of non-zero elements in k, which is strictly larger than S when the field k 
is imperfect. In the general case one can show that the G{k) orbits on S' = {G /Gy){k) are in bijection 
with the kemel of the map 7/- : H^f{k, Gy) — )• H^f{k, G) in flat (fppf) cohomology. In our example, we 
get a bijection of these orbits with the flat cohomology group Hj(k, Hp) = k* /k*^, as Hj{k, Gm) = 1- 
The semi-simple orbits in the three representations that we will study in this paper all have smooth 
stabilizers Gy. Hence we only consider the map 7 in Galois cohomology. 

3 Some representations of the split odd special orthogonal group 

Let A; be a field, with char(fc) 7^ 2. Let n > 1 and let be a fixed non-degenerate, split orthogonal 
space over k, of dimension 2?7, + 1 > 3 and determinant (—1)" in k* /k*"^. Such an orthogonal space is 
unique up to isomorphism. If {v, w) is the bilinear form on W, then we may choose an ordered basis 
{ei, 62, ... , e„, u, /„, . . . , f2, fi} of W over k with inner products given by 

{euej) = (fijj) = {ei,u) = {fi,u) = 0, 

{ei,fj)=Sij, (2) 
{u, u) = 1. 

The Gram matrix of the bilinear form with respect to this basis (which we will call the standard basis) 
is an anti-diagonal matrix. (A good general reference on orthogonal spaces, which gives proofs of these 
results, is [16].) 

Let T : W ^ W he a /c-linear transformation. We define the adjoint transformation T* by the 
formula 

{Tv, w) = (f , T*w). 

The matrix of T* in our standard basis is obtained from the matrix of T by reflection around the 
anti-diagonal. In particular, we have the identity det(T) = det(T*). We say a linear transformation 
g : W ^ W is, orthogonal if {gv, gw) = {v, w). Then g is invertible, with g~^ = g*, and det(g) = ±1 
in k*. We define the special orthogonal group SO(W^) of W by 

SO{W) := {g e GL{W) : gg* = g*g = 1, det(^7) = 1}. (3) 

We are going to consider the arithmetic invariant theory for three representations V of the reductive 
group G = SO(W) over k. 

The first is the standard representation V = W, which is irreducible and symmetrically self-dual 
(isomorphic to its dual by a symmetric bilinear pairing) of dimension 2n + 1. Here we will see that 
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the invariant polynomial q2{v) := {v, v) generates the ring of polynomial invariants and separates the 
non-zero orbits over k. 

The second is the adjoint representation V = 3o{W), which is irreducible and symmetrically 
self-dual of dimension + n. This representation is isomorphic to the exterior square A^iW) of W, 
and can be realized as the space of skew self-adjoint operators: 

V = a\W) = {T:W ■.T = -T*}, (4) 

where g E G acts by conjugation: T gTg^^ = gTg*. The Lie bracket on V is given by the formula 
[Ti, T2] = T1T2 - TaTi and the duality by (Ti, T2) = Trace(TiT2). Here the theory of G'(A;)-orbits in 
a fixed G'(A;'^) -orbit is the Lie algebra version of stable conjugacy classes for the group G = SO{W). 

The third is a representation V which arises in Vinberg's theory, from an outer involution 9 of the 
group GL{W). It is isomorphic to the symmetric square Sym^(iy) of W, and can be realized as the 
space of self-adjoint operators: 

V = Sym\W) = {T:W ■.T = T*}, (5) 

where again G = SO(W^) acts by conjugation. This representation has dimension + 3n+ 1 and is 
symmetrically self-dual by the pairing (Ti, T2) = Trace(TiT2). We will see that there are stable orbits, 
and that the arithmetic invariant theory of the stable orbits involves the arithmetic of hyperelliptic 
curves of genus n over k, with a /c-rational Weierstrass point. 

We note that the third representation V is not irreducible, as it contains the trivial subspace 
spanned by the identity matrix, and has a non-trivial invariant linear form given by the trace. When 
the characteristic of k does not divide 2n + 1 = dim(Vr), the representation V is the direct sum of the 
trivial subspace and the kernel of the trace map, and the latter is irreducible and symmetrically self-dual 
of dimension + 3n. When the characteristic of k divides 2n + 1 the trivial subspace is contained 
in the kernel of the trace. In this case V has two trivial factors and an irreducible factor of dimension 
2n^ + 3n — 1 in its composition series. 

4 Invariant polynomials and the discriminant 

In the standard representation V = WofG = SO(l^), the quadratic invariant q2iv) = {v, v) generates 
the ring of invariant polynomials. We define A = ^2 in this case. When A(t>) 7^ 0, the stabilizer G„ is 
the reductive subgroup SO(t/), where U is the hyperplane in W of vectors orthogonal to v. 

In the second and third representations, the group SO(H^) acts by conjugation on the subspace V 
of End(W^). Hence the characteristic polynomial of an operator T is an invariant of the G(A;)-orbit. 

For the adjoint representation, the operator T is skew self-adjoint and its characteristic polynomial 
has the form 

f{x) = det{xl -T) = x^'^+i + C2x2"-' + C4x2"-3 + . . . + C2nx = xg{x^). 

with coefficients C2m E k. The coefficients C2m are polynomial invariants of the representation, with 
deg(c2m) = 2m. These polynomials are algebraically independent and generate the full ring of poly- 
nomial invariants on V" = 3o{W) over k [3, Ch 8, §8.3, §13.2, VI].. An important polynomial invariant, 
of degree 2n(2n + 1), is the discriminant A of the characteristic polynomial of T: 

A = A(c2, C4, . . . , C2„) = disc f{x). 
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This is non-zero in k precisely when the polynomial /(x) is separable, so has 2n + 1 distinct roots in 
the separable closure of k. The condition A(T) 7^ defines the regular semi-simple orbits in the Lie 
algebra. For such an orbit, we will see that the stabilizer Gt is a maximal torus in G, of dimension n 
over k. 

For the third representation V on self-adjoint operators, the characteristic polynomial f{x) of T 
can be any monic polynomial of degree 2n + 1; we write 

f{x) = det(x/ - T) = X^'^+l + CiX^'^ + C2X2"-1 + ■ ■ • + C2nX + C2n+l 

with coefficients Cm E k. Again the Cm give algebraically independent polynomial invariants, with 
deg(cm) = m, which generate the full ring of polynomial invariants on V over k. The discriminant 

A = A(ci, C2, . . . , C2n+i) = disc /(x) 

is defined as before, and is non-zero when f{x) is separable. We will see that the condition A(T) 7^ 
defines the stable orbits of G on V. For such an orbit, we will see that the stabilizer Gt is a finite 
commutative group scheme of order 2^" over k, which embeds as a Jordan subgroup scheme of G 
(see [12, Ch 3]). 

5 The orbits with non-zero discriminant 

In this section, for each of the three representations V, we exhibit an orbit for G where the invariant 
polynomials described above take arbitrary values in k, subject to the single restriction that A 7^ 0. 
We calculate the stabilizer Gy and its cohomology H^{k, G^,) in terms of the values of the invariant 
polynomials on v. We also give an explicit description of the map 7 : H^{k, Gy) — )■ H^{k, G). We 
note that all three representations arise naturally in Vinberg's invariant theory, and the representative 
orbits that we will construct are in the Kostant section (cf. [18]). 

When V = W is the standard representation, let d be an element of k*. The vector v = ei + dfi 
has q2{v) = A(t>) = d. The stabilizer Gy acts on the orthogonal complement U of the non-degnerate 
line kv in W , which is a quasi-split orthogonal space of dimension 2n and discriminant d in k* /k*"^. 
(The discriminant oi an orthogonal space of dimension 2n is defined as (—1)" times its determinant.) 
This gives an identification Gy = SO(f/), where the special orthogonal group S0([/) is quasi-split over 
A; and split by A; (v^). Witt's extension theorem [16, Ch 1] shows that all vectors with g2(w^) = lie in 
the G(A;)-orbit of v, so the invariant polynomials separate the orbits over k with non-zero discriminant. 
One can also show that there is a single non-zero orbit with q2{v) = 0, represented by the vector 
V = ei=ei + 0/1. 

The cohomology set H^{k,SO{U)) classifies non-degenerate orthogonal spaces U' of dimen- 
sion 2n and discriminant d over k, and the cohomology set H^{k, SO(W^)) classifies non-degenerate 
orthogonal spaces W of dimension 2n + 1 and determinant (—1)" over k, with the trivial class corre- 
sponding to the split space W. The map 

7 : H\k, Gy) = H\k, SO{U)) H\k, G) = H\k, SO{W)) 

is given explicitly by mapping the space f/' to the space W^' = U'+{d). Witt's cancellation theorem [16] 
shows that the map 7 is an injection of sets in this case, so the arithmetic invariant theory for the 
standard representation of any odd orthogonal group is the same as its geometric invariant theory. 
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For the second representation V = so{W) = A'^{W), let 

f(x) = + C2X2"-1 + C4x2"-3 + . . . + C2nX 

be a polynomial in k[x] with non-zero discriminant. We will construct a skew self-adjoint operator T 
on W with characteristic polynomial f{x). Since f(x) = xh{x) = xg{x^), we have 

disc/(x) = disc = (— 4)'^C2„ disc5f(x)^. 

Let K = k[x]/ {g{x)), E = k[x]/{h{x)), and L = k[x]/{f{x)). By our assumption that A 7^ 0, these 
are etale A;-algebras of ranks n, 2n, and 2n + 1 respectively. We have L = E ® k. Furthermore the 
map X — —x induces an involution r of the algebras E and of L, with fixed algebras K and K ® k 
respectively. 

Let /3 be the image of x in L = k[x\/{f{x)), so f{j3) = in L and is a unit in L*, We 
define a symmetric bilinear form ( , ) on the fc- vector space L = k + k(3 + k/3'^ -\ — ■ + A;/3^" by taking 

(A, /i) := the coefficient of /J^*^ in the product (— l)'^A/i'^. (6) 

This is non-degenerate, of determinant (—1)", and the map t(A) = (3\ is skew self-adjoint, with char- 
acteristic polynomial f{x). Finally, the subspace M = k+k(3+- ■ ■ + is isotropic of dimension n, 
so the orthogonal space L is split and isomorphic to W over k. Choosing an isometry 9 : L ^ W we 
obtain a skew self-adjoint operator T = 9t9^^ on W with the desired separable characteristic polyno- 
mial. Since the isometry 9 is unique up to composition with an orthogonal transformation of W, the 
orbit of T is well-defined. The stabilizer of T in 0{W) has A;-points {\ e L* : A^+^ = 1}. The sub- 
group Gt which fixes T is a maximal torus in G = SO{W), isomorphic to the torus ResK/k Ui{E/K) 
of dimension n over k. 

Over the separable closure k^ of k, any skew self-adjoint operator S with (separable) character- 
istic polynomial f{x) is in the same orbit of T. Indeed, since f{x) is separable, it is also the minimal 
polynomial of T and S, so we can find an element g in GL(H^) with S = gTg~^. Since both operators 
are skew self-adjoint, the product g*g is in the centralizer of T in GL(iy). The centralizer of T in 
End(Vr) is the algebra k[T] = L. Since g*g is self-adjoint in L*, and its determinant is a square in k*, 
we see that g*g is an element of the subgroup K* x k*'^. Over the separable closure, every element of 
K* X k*"^ is a norm from L*: g*g = h^^'^. Then gh~^ is an orthogonal transformation of W over k^ 
mapping T to S. Hence S is in the SO(iy)(A;")-orbit of T. 

To understand the orbits with a fixed separable characteristic polynomial over k, we need an 
explicit form of the map 7 in Galois cohomology. Since the stabilizer of T is abelian, the pointed set 
H^{k, Gt) is an abelian group, which is isomorphic to K* /NE* by Hilbert's Theorem 90. The map 

7 : K*/NE* = H\k, Gt) H\k, G) = H\k, SO{W)) 

is given explicitly as follows. We first associate to an element k E K* the element a = (k, 1) in 
(L^)* = K* X k*, with square norm from L* to k*. We then associate to a the vector space L with 
symmetric bilinear form 

(A, fi)a ■= the coefficient of in the product ( — l)"aA/i'^. (7) 

This orthogonal space has dimension 2n + 1 and determinant (—1)" over k, and its isomorphism 
class depends only on the class of k in the quotient group K* /NE* = H^{k, Gt)- 
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Lemma 3 The orthogonal space represents the class 7(k) in H^{k, SO(W)). 

Proof: We first recall the recipe for associating to a cocycle g^^ on the Galois group with values in 
SO{W){k'') a new orthogonal space W over k. We use the inclusion SO(W) GL(W^) and the 
triviality of H^{k, GL{W)) to write = h'^h" for an element h e GL{W){k''). We then define a 
new non-degenerate symmetric bilinear form on W by the formula 

{v,wy = {h^^v,h~^w). (8) 

This takes values in k and defines the space W, which has dimension 2n + 1 and determinant (—1)". 
The isomorphism class of W over k depends only on the cohomology class of the cocycle g^^ in 

H\k,SO{W)). 

In our case, the cocycle g^r representing 'j(k) comes from a cocycle with values in the stabilizer 
Gy. This is a maximal torus in SO{W), which is a subgroup of the maximal torus Resi/kGm of 
GL(H^). This torus already has trivial Galois cohomology, so we can write g^r = /h with h e 
{L®k^)* satisfying h^^'^ = a. Substituting this particular h into formula (8) for the new inner product 
on W completes the proof. □ 

We note that the class k above will be in the kemel of 7 precisely when the quadratic space W 
with bilinear form ( , )q is split. Such classes give additional orbits of SO(W^) on 5o{W) = A'^{W) 
over k with characteristic polynomial f{x). 

The analysis for the third representation V = Sym^ ( VT) is similar. Here we start with an arbitrary 
monic separable polynomial f(x) = a;^""^^ + cix^" + ■ ■ ■ + C2n+i and wish to construct a self-adjoint 
operator T on W with characteristic polynomial f{x). We let L = k[x]/{f{x)), which is an etale 
A;-algebra of rank 2n + 1, and let (3 be the image of x in L. We define a symmetric bilinear form (A, fi) 
on L = k + k(3 + ■■■ + k(3'^"' by taking the coefficient of in the product A/i. This is non-degenerate 
of determinant (—1)", and the map t{\) = (3\ is self-adjoint, with characteristic polynomial f{x). 
Finally, the subspace M = k + k(3 + ■ ■ ■ + k(3"~^ is isotropic of dimension n, so the orthogonal space 
L is split and isomorphic to W over k. Choosing an isometry 9 : L ^ W, we obtain a self-adjoint 
operator T = OtO^^ on W with the desired separable characteristic polynomial. Since the isometry 6 is 
unique up to composition with an orthogonal transformation of W , the orbit of T is well-defined. The 
stabilizer of T in 0( W) has A;-points {A G L* : A^ = 1}. The subgroup Gt in SO(M^) which fixes T 
is the finite etale group scheme A of order 2^", which is the kemel of the norm map YlesL/kil^i) 1^2- 

Over the separable closure A;"* of k, any self-adjoint operator S with (separable) characteristic 
polynomial f(x) is in the same orbit as T. Indeed, since f{x) is separable, it is also the minimal 
polynomial of T and S, so we can find an element g e GL(IV) with S = gTg^^. Since both operators 
are self-adjoint, the product g*g is in the centralizer of T in GL(iy). The centralizer of T in End(H^) 
is the algebra k[T] = L, so g*g is an element of L*. Over the separable closure, every element of L* is 
a square: g*g = h?. Then gh"^ is an orthogonal transformation of W over k'^ mapping T to S. Hence 
S is in the SO(Vr) (A;^)-orbit of T. 

We now consider the orbits with a fixed separable characteristic polynomial over k. Since the 
stabilizer of T is again abelian, the pointed set H^{k, Gt) is an abelian group which is isomorphic to 
7v=i by Kummer theory. The map 

7 : H\k,GT) = {L*/L*^)n=i H\k,G) = H\k,SO{W)) 
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is given explicitly as follows. We associate to an element a in (L*)Ar=i the orthogonal space L with 
bilinear form (A, /i)^ given by the coefficient of in the product aXfi. This orthogonal space has 
dimension 2n + 1 and determinant (—1)" over k. Its isomorphism class over k depends only on the 
image of a in the quotient group {L* / L*'^)n=i = H^{k,GT)- This orthogonal space represents the 
class 7(a) in H^{k, SO{W)). The proof is the same as that of Lemma 3. We first observe that the 
map taking the cocycle from Gt to SO(W^) to GL(W^) can also be obtained by mapping Gt to the 
maximal torus Resi/k Gm in GL(H^). This torus has trivial cohomology, so a = h'^ with h E (L® A;'')*, 
and this choice of h gives the inner product (A, /i)^. The class a will be in the kernel of 7 precisely 
when the quadratic space L with bilinear form ( , is split; such classes give additional orbits of 
SO{W) on Sym^{W) over k with characteristic polynomial f{x). 

We summarize what we have established for the representations V = so{W) and V = Sym^{W). 

Proposition 4 For each monic separable polynomial f{x) of degree 2n + l over k of the form f{x) = 
xg{x'^) there is a distinguished SO {W){k) -orbit of skew self-adjoint operators T on W with char- 
acteristic polynomial f{x). All other orbits on /\^{W) with this characteristic polynomial lie in 
the SO {W){k'') -orbit of T, and correspond bijectively to the non-identity classes in the kernel of 
7 : K*/NE* H\k,SO{W)), where K = k[x]/{g{x)) and E = k[x\/ {g{x^)). 

For each monic separable polynomial f{x) of degree 2n + 1 over k there is a distinguished 
SO(W) {k) -orbit of self -adjoint operators T onW with characteristic polynomial f{x). All other orbits 
on Sym^(iy) with this characteristic polynomial lie in the SO {W){k'^) -orbit ofT, and correspond 
bijectively to the non-identity classes in the kernel of •j : {L*/L*'^)n=i — ?■ H^{k,SO{W)), where 
L = k[x]/{f{x)). 

6 Stable orbits and hyperelliptic curves 

For both representations V = A'^(W) and V = Sjm^iW) of G = SO{W) we associated to the 
distinguished orbit T with separable characteristic polynomial f{x) and any class a in the cohomology 
group H^{k,GT) a symmetric bilinear form (A,/x)q on the /c-vector space L = k[x]/{f{x)). The 
class a is in the kemel of the map 7 : H^{k, Gt) H^{k, G) precisely when this quadratic space is 
split over k. However, exhibiting specific classes a 7^ 1 where this space is split is a difficult general 
problem, so it is difficult to exhibit other orbits with this characteristic polynomial. 

In the case of the third representation V = Sym^(W^), the orbits T with A(T) 7^ are stable; 
namely, they are closed (defined by the values of the invariant polynomials over the separable closure) 
and have finite stabilizer (the commutative group scheme A = ResL/kil^2)N=i of order 2^"). In this 
case, we will use some results in algebraic geometry, on hyperelliptic curves with a Weierstrass point 
and the Fano variety of the complete intersection of two quadrics in P(L©fc), to produce certain classes 
in the kemel of the map 7 : H\k,A) H\k,SO{W)). 

Let G be the smooth projective hyperelliptic curve of genus n over k with affine equation = 
f{x) and /c -rational Weierstrass point P above x = 00. The functions on G which are regular outside 
of P form an integral domain: 

H\G - P,0c-P) = k[x,y]/iy' = fix)) = k[x, vTM]- 

The complete curve G is covered by this affine open subset Ui, together with the affine open 
subset U2 associated to the equation = t;^""'~^/(l/t>) and containing the point P = (0,0). The 
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gluing of Ui and U2 is by {v,w) = y/x""*"^) and {x,y) = {l/v,w/v^^^) wherever these maps 
are defined. Let J denote the Jacobian of C over k and let J [2] the kernel of multiplication by 2 on J. 
This is a finite etale group scheme of order 2^" over k. 

Lemma 5 The group scheme J [2] of 2-torsion on the Jacobian ofC is canonically isomorphic to the 
stabilizer A = ResL/k{fi2)N=i of the orbit T in SO{W). 

Proof: Write L = k[x]/{f{x)) = k + k/S -{ h where = 0. The other Weierstrass points 

P,, = ivif^)^ 0) of C{k'^) correspond bijectively to algebra embeddings r] : L ^ k^. Associated to such 
a point we have the divisor = (P^) — (P) of degree zero. The divisor class of lies in the 2-torsion 
subgroup J[2]{k'^) of the Jacobian, as 

2dri = div(x — ri{f3)). 

The Riemann-Roch theorem shows that the classes d^j generate the finite group J[2]{k'^), and satisfy 
the single relation 

^(rf,) = div(y). 

Since the Galois group of k^ acts on these classes by permutation of the embeddings 77, we have an 
isomorphism of group schemes: J[2] = ResL/fc(/i2)//i2- This quotient of ResL/fc(/i2) is isomorphic to 
the subgroup scheme A = ResL/kif^2) n=i, as the degree of L over k is odd. This completes the proof. 

□ 

The exact sequence of Galois modules, 

^ J[2](F) J(F) ^ J{k') ^ 0, 

gives an exact descent sequence 

^ J{k)/2J{k) H\k, J[2]) H\k, J)[2\ 

in Galois cohomology. By Lemma 5, the middle term in this sequence can be identified with the group 
(/c, A) = [k, Gt), and our main result in this section is the following. 

Proposition 6 The subgroup J{k) /2J{k) of H^{k, A) = H^{k,GT) lies in the kernel of the map 

7 : H\k,GT) H\k,G). 

Proof: We first make the descent map from H^{k, A) to H^{k, J) [2] more explicit. That is, we need to 
associate to a class a in the group 

H\k,A) = iL*/L*^)j,=, 

a principal homogeneous space Fa of order 2 for the Jacobian J over k. The class a will be in the 
subgroup J{k) /2J{k) precisely when the homogeneous space has a A;-rational point. 

We have previously associated to the class a the orthogonal space L with symmetric bilinear form 
(A, 11) a '■= the coefficient of in the product aA/i. We also defined a self-adjoint operator given by 
multiplication by /3 on L, and that gives a second symmetric bilinear form on L: {I3X, /i)^ = (A, I3^)a- 
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Let M = L ® k, which has dimension 2n + 2 over k, and consider the two quadrics on M given 

by 

g(A,a) = (A,A)„ 
Q'(A,a) = (/3A,A)« + al 

The pencil uQ — vQ' is non-degenerate and contains exactly 2n + 2 singular elements over k^, namely, 
the quadric Q at = and the 2n + 1 quadrics r]((3)Q — Q' at the points where f{ri{f3)) = 0. 
Hence the base locus is non-singular in P(A/) and the Fano variety of this complete intersection, 
consisting of the ra-dimensional subspaces Z of M which are isotropic for all of the quadrics in the 
pencil, is a principal homogeneous space of order 2 for the Jacobian J (c.f. [7]). More precisely, there 
is a commutative algebraic group la with 2 components over k, having identity component J and 
non-identity component Fa. 

Since the discriminant of the quadric uQ — vQ' in the pencil is equal to t>^""'"^/(x) with x = u/v, 
a point c = (x, y) on the hyperelliptic curve y"^ = f{x) determines both a quadric = xQ — Q' in the 
pencil together with a ruling of Q^, i.e., a component of the variety of (?i + l)-dimensional Qa;-isotropic 
subspaces in M. Each point gives an involution of the corresponding Fano variety 9{c) : Fa ^ Fa with 
2^" fixed points over a separable closure k^ of k. The involution 9{c) is defined as follows. A point of 
Fa consist of a common isotropic subspace Z of dimension nin M k'^. The point c gives a maximal 
isotropic subspace Y for the quadric which contains Z. If we restrict any non-singular quadric in 
the pencil (other than Q^) to Y, we get a reducible quadric which is the sum of two hyperplanes: Z and 
another common isotropic subspace Z'. This defines the involution: 9{c){Z) = Z'. In the algebraic 
group la, we have that Z + Z' is, the class of the divisor (c) — (P) of degree zero in J. 

Now assume that the class a is in the subgroup J{k) /2J{k). Then its image in H^{k, J) is trivial, 
and the homogenous space Fa has a /c-rational point. Hence there is a fc-subspace Z of M = L (B k 
which is isotropic for both Q and Q'. Since it is isotropic for Q', the subspace Z does not contain the 
line © A;, so its projection to the subspace L has dimension n and is isotropic for Q. This implies 
that the orthogonal space L with bilinear form (A, u)a is split, so the class a is in the kernel of the map 
7 : H\k, A) H\k, SO(W^)). □ 

Note that when c = P, the Weierstrass point over x = oo, the involution 9{P) is induced by 
the linear involution (A, a) — )■ (A, —a) of M = L (B k. The fixed points are just the ?i-dimensional 
subspaces X over k'^ which are isotropic for both quadrics 

g(A) = (A,A)„, 
g'(A) = {(3\,\)a 

on the space L of dimension 2n + 1 over k. There are 2^" such isotropic subspaces over k'^, and they 
form a principal homogeneous space for J [2]. The variety Fa has a A;-rational point when a lies in the 
subgroup J{k) /2J{k), but only has a A; -rational point fixed by the involution 9{P) when a is the trivial 
class in H^{k,J[2]). 

Remark 7 The finite group scheme A = J[2] does not determine the hyperelliptic curve C over k. 
Indeed, for any class d E k*/k*'^, the hyperelliptic curve Cd with affine equation dy"^ = f{x) has 
the same 2-torsion subgroup of its Jacobian. This Jacobian Jd of Cd acts on the Fano variety of the 
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complete intersection of the two quadrics given by 



Q{X,a) = {X,\)a, 
Q'(A,a) = {l3X,X)a + da^. 

Indeed, the discriminant of the quadric uQ — vQ' in the pencil is equal to dv'^"'~^'^f{x), where x = u/v. 
A similar argument then shows that the subgroup Jd{k) /2Jd{k) is also contained in the kernel of the 
map 7 on H^{k, A). 

7 Arithmetic fields 

In this section, we describe the orbits in our three representations when A; is a finite, local, or global 
field. 

7.1 Finite fields 

First, we consider the case when k is finite, of odd order q. In this case, H^{k, SO (14^)) = 1 by Lang's 
theorem, as SO(iy) is connected. As a consequence, every quadratic space of dimension 2/2 + 1 and 
determinant (—1)" is split, and all elements of H^{k, Gt) lie in the kernel of 7. 

In the standard representation V = W the stabilizer of a vector v with ^2(^0 7^ is the connected 
orthogonal subgroup SO(t/), which also has trivial first cohomology. So for every non-zero element d 
in k*, there is a unique orbit of vectors with q2iv) = d. (We have already seen this for general fields 
via Witt's extension theorem.) 

In the adjoint representation V = 5o(W), the stabilizer of a vector T with A(T) 7^ is the 
connected torus Resx/k Ui{E/K), which also has trivial first cohomology. So for each separable char- 
acteristic polynomial of the form f{x) = xg{x'^) there is a unique orbit of skew self-adjoint operators 
T with characteristic polynomial f{x). 

In the representation V = Sym^{W) the stabilizer of T with characteristic polynomial f{x) 
satisfying disc(/) = A(T) 7^ is the finite group scheme A = (ResL/k f'2)N=i- In this case 
H^{k,A) = {L* / L*'^)iy=i is an elementary abelian 2-group of order 2"*, where m + 1 is the num- 
ber of irreducible factors of f{x) in k[x]. So 2"* is the number of distinct orbits with characteristic 
polynomial f{x). But this is also the order of the stabilizer H^{k, A) = A{k) = (L*[2])Ar=i of any 
point in the orbit. Hence the number of self-adjoint operators T with any fixed separable polynomial is 
equal to the order of the finite group SO{W){q). This is given by the formula 

# SO{W){q) = - l)(g2n-2 _iy..^q2_i^_ 

By Lang's theorem, we also have H^{k, J) = 0, where J is the Jacobian of the smooth hyperel- 
liptic curve y"^ = f{x) of genus n over k. Hence the homomorphism J{k) /2J{k) — )■ H^{k, A) is an 
isomorphism and every orbit with characteristic polynomial f{x) comes from a A; -rational point on the 
Jacobian. 

7.2 Non-archimedean local fields 

Next, we consider the case when k is a non-archimedean local field, with ring of integers O and finite 
residue field 0/nO of odd order. In this case, Kneser's theorem on the vanishing of for simply- 
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connected groups (cf. [19, Th. 6.4], [21]) gives an isomorphism 

H\k,SO{W)) ^ H\k,fi2) = (Z/2Z). 

For the standard representation V = W,we also have H\k, Gy) = H\k, SO(f/)) = (Z/2Z), 
except in the case when dim(V") = 3 and q2{v) = 1, when SO(f/) is a split torus and H^{k, SO(f/)) = 
1. The map 7 is a bijection except in the special case. 

For the adjoint representation V = so{W), Kottwitz has shown in the local case that the map 

7 : H\k,Gy) = (IC/NE*) H\k,G) = (Z/2Z) 

is actually a homomorphism of groups [10]. Let f{x) = xg{x^), so K = k[x]/{g{x)) and E = 
k[x]/{g{x'^)). It follows from local class field theory that the group K* /NE* is elementary abelian 
of order 2™, where m is the number of irreducible factors gi{x) of g{x) such that gi{x'^) remains 
irreducible over k. Kottwitz also shows that that the map 7 is surjective when m > 1. Hence the 
number of orbits with separable characteristic polynomial f{x) is 1 when m = 0, and is 2"*~^ when 
m > 1. 

For the third representation V = Syra^{W), the map 

7 : H\k, A) = H\k, J[2]) ^ H\k, ^2) = (Z/2Z) 

is an even quadratic form. The associated bilinear form is the cup product on H^{k, J[2]) induced from 
the Weil pairing J[2] x J[2] — 112, and J{k) /2J{k) is a maximal isotropic subspace on which 7 = 0. 
This allows us to count the number of stable orbits with a fixed characteristic polynomial. 

Let m + 1 be the number of irreducible factors of /(x) in k\x\, and let Ol be the integral closure 
of the ring O in L. Then H^{k, A) = (L*/L*^) 7v=i has order 2^™ and the number of stable orbits with 
characteristic polynomial f{x) is equal 2™-i(2" + 1) = 2^"^-^ + 2™-^. The subgroup J{k) /2J{k) has 
order 2™, which is also the order of the subgroup (02/ O}^) n=i of units. These two subgroups coincide 
when the polynomial f(x) has coefficients in O and the quotient algebra 0[x]/ (/(x)) is maximal in L. 

7.3 The local field M 

We next consider the orbits in our representations when A; = M is the local field of real numbers. 
Then the pointed set H^{k, G) = H^{k, '^0{W)) has n + \ elements, corresponding to the quadratic 
spaces W of signature (p, q) satisfying: p + q = 2n + 1 and q = n (mod 2). The pointed set 
H^{k, Gy) = H^{k, SO(f/)) for the standard representation has n + 1 elements when q2{v) has sign 
(— 1)", and has n elements when q2{v) has sign —(—1)". The map 7 is a bijection in the first case 
and an injection in the second case, when the definite quadratic space W does not have an orbit with 
q2{w*) = q2{v). 

In the second and third representations, H^{k, Gt) is an elementary abelian 2-group, and we will 
consider the situations where it has maximal rank. For the adjoint representation V = so(W), this 
occurs when all of the nonzero roots of the characteristic polynomial f{x) of the skew self-adjoint 
transformation T are purely imaginary. Thus /(x) = xg{x'^) where g{x) factors completely over 
the real numbers and all of its roots are strictly negative. In this case, the 2-group H^{k,GT) = 
K* /NE* = (M*)"/A^(C*)'^ has rank n. The real orthogonal space W decomposes into n orthogonal 
T-stable planes and an orthogonal line on which T = 0. The signatures of these planes determine the 
real orbit of T. Writing n = 2m or n = 2m + 1, we see that there are (^) elements in the kernel of 
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7- One can show that 7 is surjective in this case, and calculate the order of each fiber as a binomial 
coefficient (^). 

For the symmetric square representation V = Sym^(H^), the 2-group H^{k,GT) has maximal 
rank when the characteristic polynomial /(x) of the self-adjoint transformation T factors completely 
over the real numbers. In this case, H^{k,GT) = {{R*f"^^/{R*^f''+^)N=i has rank 2n. The real 
orthogonal space W decomposes into 2n + l orthogonal eigenspaces for T, and the signatures of these 
lines determine the real orbit. Hence there are (^"^^) elements in the kernel of 7. One can also show 
that 7 is surjective in this case, and calculate the order of each fiber as a binomial coefficient (^"^^) 
with k = n (mod 2). 

7.4 Global fields 

Finally, we consider the representation Sym^(\^) when is a global field. In this case, the group 
H^{k, A) = H^{k, J[2]) is infinite. We will now prove that there are also infinitely many classes in the 
kernel of 7, so infinitely many orbits with characteristic polynomial /(x). 

Proposition 8 Every class a in the 2-Selmer group Sel(J/A;, 2) of H^{k, J[2]) lies in the kernel of'j, 
so corresponds to an orbit over k. 

Proof: By definition, the elements of the 2-Selmer group Sel( J/fc, 2) correspond to classes in H^{k, J[2]) 
whose restriction to H^{ky, J[2]) is in the image of J{ky) /2J{kv) for every completion ky. Hence the 
orthogonal space associated to the class ^{cty) in H^{kv, SO{V)) is split at every completion k^. 
By the theorem of Hasse and Minkowski, a non-degenerate orthogonal space U of dimension 2n + 1 is 
split over k if and only if Uy = U k^ is split over every completion ky. Hence the orthogonal space 
U associated to 7(a) is split over k, and a lies in the kemel of 7. □ 

The same argument applies to the Selmer group of the Jacobian Jd of the hyperelliptic curve 
dy'^ = f{^)^ for ^ny class d E k* /k*"^. Since the 2-Selmer groups of the twisted curves are known to 
become arbitrarily large (cf. [5] for the case of genus n = 1), the number of A;-rational orbits is infinite. 

8 More general representations 

The three representations V of SO(W^) that we have studied illustrate various phenomena which occur 
in many other cases. For the standard representation, we have seen that the invariant polynomial q2 
distinguishes the orbits with A 7^ over any field k. Here the arithmetic invariant theory is the same 
as the geometric invariant theory. 

This pleasant situation also occurs for orbits where the stabilizer Gy is trivial! An interesting 
example for the odd orthogonal group SO(W^) is the reducible representation V = W (B A'^{W). 
This occurs as the restriction of the adjoint representation of the split even orthogonal group of the 
space W © ( — I)- In this representation, the vector v = {w,T) is stable if and only if the 2n + 1 
vectors {w, T{w),T'^{w), . . . , T^'^(w)} form a basis of W, or equivalently, if the invariant polynomial 
A{v) = det{{T'{w),T^{w))) is non-zero. In this case Gy = 1. 

One complication in this case is that the A;-orbits do not cover the A; -rational points of the cate- 
gorical quotient: the map on points 

V{k)/SO{W){k) {V/SO{W)){k) 



14 



is not surjective. This situation is far more typical in invariant theory than the surjectivity for the three 
representations we studied. Another atypical property of the three (faithful) representations we studied 
was that a generic vector had a nontrivial stabilizer. For a generic v ma typical faithful representation 
V" of a reductive group G, the stabilizer Go is trivial. For G a torus and k complex, Gy is always the 
kernel of the representation; meanwhile, for G simple, there are only finitely many exceptions (see [20, 
p. 229-235]). 

The adjoint representations V = Qof split reductive groups G generalize the second representa- 
tion V = A^(iy) = 5o{W). Here the invariant polynomials correspond to the invariants for the Weyl 
group on a Cartan subalgebra, and generate a polynomial ring of dimension equal to the rank of G. 
The orbits where the discriminant A is non-zero correspond to the regular semi-simple elements in g, 
and the stabilizer G^ of such an orbit is a maximal torus in G. As an example, one can take the adjoint 
representation V = Sym^(iy) of the adjoint form PGSp(Vr) = PGSp2„ of the symplectic group, 
where the degrees of the invariants are 2, 4, 6, ... , 2n. [3, Ch 8, §13.3, VI]. For some applications to 
knot theory, see [15]. 

The representations which occur in Vinberg's theory for torsion automorphisms 9 generalize 
the third representation V = Sym^(iy). Here the invariants again form a polynomial ring. As an 
example, one can take the reducible representation A'^{W) of the group PGSp(lV) = PGSp2„, which 
corresponds to the pinned outer involution 9 of PGL2n- When 9 lifts a regular elliptic class in the 
Weyl group, the orbits where the discriminant A is non-zero are stable, and the stabilizer Gy is a finite 
commutative group scheme over k. Several examples of this type were discussed in [8] and [1]. 

9 Integral orbits 

In order to develop a truly complete arithmetic invariant theory, we should consider orbits in repre- 
sentations not just over a field, but over Z or a general ring. The descent from an algebraically closed 
field to a general field that we have discussed in Sections 2-8 gives an indication of some of the issues 
that arise over more general rings, and it serves as a useful guide for the more general integral theory. 
In particular, just as a single orbit over an algebraically closed field can split into several orbits over a 
subfield, an orbit over say the field Q of rational numbers may then split into several orbits over Z. 

Often some of the most interesting arithmetic occurs in the passage from Q to Z. For example, 
consider the classical representation given by the action of SL2 on binary quadratic forms Sym2(2). As 
we have already noted, an orbit over Q (as over any field) is completely determined by the value of the 
discriminant d of the binary quadratic forms in that orbit. However, the set of primitive integral orbits 
inside the rational orbit of discriminant d G Z does not necessarily consist of one element, but rather is 
in bijection with the set of (oriented) ideal classes of the quadratic order Z[((i + \Q) / 2] in the quadratic 
field QiVd) (see, e.g., [6]). 

In general, to discuss integral orbits we must fix an integral model of the representation being 
considered. We give some canonical integral models for the three representations we have studied. For 
the first representation, we take W to be the odd unimodular lattice of signature (n + l,n) defined 
by (2). Because this lattice is self-dual, we can define the adjoint of an endomorphism of W over Z. 
The group G is then the subgroup of GL(iy) consisting of those transformations g such that gg* = 1 
and det{g) = 1. This defines a group that is smooth over Z[l/2] but is not smooth over Z2. For 
the other representations of G, we define A2{W) as the lattice of skew self-adjoint endomorphisms of 
W equipped with the action of G by conjugation; we similarly define Sym2(H^) to be the lattice of 
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self-adjoint endomorphisms of W. Our objective is to describe the orbits of G on each of these three 
G-modules, or at least those orbits where the discriminant invariant is nonzero. 

For the standard representation W, we have already seen that there is a unique orbit over Q for 
each value of the discriminant d E Q*. An invariant of a Z-orbit of a vector w in the lattice W with 
{w,w) = d is, the isomorphism class of the orthogonal complement U = (Zw)^, which is a lattice 
of rank 2n and discriminant d over Z. Although W is an odd lattice, the lattice U can be either even 
or odd. For example, when n = 3, the orthogonal complement U of a primitive vector w is an even 
bilinear space of rank 2 and discriminant d (so corresponds to an integral binary quadratic form of 
discriminant d) if and only if the vector w has the form w = ae + bv + cf with a and c even and b 
odd. In this case d = b"^ + 2ac = 1 modulo 8, and the orbits of G(Z) on such vectors form a principal 
homogeneous space for the ideal class group of the quadratic order Z[{d + \/d)/2] of discriminant d. 
We note that these are precisely the quadratic orders where the prime 2 is split. In this case the group 
G(Z) is isomorphic to the normalizer A^(ro(2)) of ro(2) in PSL2(M), and the orbits described above 
correspond to the Heegner points of odd discriminant on the modular curve Xo(2)+ [9, §1]. 

We consider next the second representation V = A2{W). Here, we find that the integral orbits 
of SO(M) on the self-adjoint transformations T : M ^ M with (separable) characteristic polynomial 
f{x) = xg{x'^) E TL\x\ correspond to data which generalize the notion of a "minus ideal class" for the 
ring R = Z[x]/(/(x)). More precisely, the ring R in L = Q[x]/ {f{x)) has an involution r sending 
(3 to —(3, where /3 denotes the image of x in R. Let us consider pairs (J, a), where / is a fractional 
ideal for R, the element a is in the Q-subalgebra F of L fixed by r, the product //"' is contained in 
the principal ideal (a), and N(I)N(F) = N(a). Such a pair (/, a) gives / the structure of an integral 
lattice having rank 2n + 1 and determinant (—1)", where the symmetric bilinear form on / is defined 
by 

(x, y) := coefficient of in (— ^xy'^. (9) 

The pair (/', a') gives an isometric lattice if /' = cl and a' = cc^a for some element c E L*. The 
operator S : I I defined by S{x) = (3x is skew self-adjoint, and has characteristic polynomial 
f{x). If the integral lattice determined by the pair (/, a) has signature (n + 1, n) over R, there is an 
isometry 9 : I M (cf. [22]), which is well-defined up to composition by an element in 0(M). We 
obtain an SO(M)-orbit of skew self-adjoint operators with characteristic polynomial f(x) by taking 
T = 6S9~^. Conversely, since a skew self-adjoint T : W ^ W gives W the structure of a torsion free 
Z[T] = i?-module of rank one, every integral orbit arises in this manner. Thus the equivalence classes 
of pairs (/, a) for the ring R = Z[x]/ (/(a;)), as defined above, index the finite number of integral orbits 
onV = A2{W) with characteristic polynomial /(x). 

Let us now consider the third representation V = Sym2{W). When dim(iy) = 3, the kemel of 
the trace map gives a lattice of rank 5, closely related to the space of binary quartic forms for PGL2. The 
integral orbits in this case were studied in [2] and [25]. In general, the integral orbits of SO(M) on the 
self-adjoint transformations T : M — )■ M with (separable) characteristic polynomial f(x) correspond 
to data which generalize the notion of an ideal class of order 2 for the order R = Z[x]/(/(x)) in the 
Q-algebra L = Q[x]/{f{x)). More precisely, we consider pairs (/, a), where / is a fractional ideal for 
R, the element a lies in L*, the square P of the ideal / is contained in the principal ideal (a), and the 
square of the norm of / satisfies N(iy = N(a). Then the lattice / has the integral symmetric bilinear 
form 

{x, y) := coefficient of Z?^"^ in a~^xy (10) 
of determinant (—1)", and self-adjoint operator given by multiplication by /3, where (3 again denotes 
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the image of x in R. The pair (/', a') gives an isometric lattice if /' = cl and a' = (?(x for some 
element c E L*. When this lattice has signature (n + 1, n) over M, it is isometric to M and we obtain 
an integral orbit with characteristic polynomial f{x). Conversely, since a self-adjoint T : W W 
gives W the structure of a torsion free Z[T] = i?- module of rank one, every integral orbit arises in this 
way. Thus pairs (/, a) for the ring R = Z[x]/ {f{x)), up to the equivalence relation defined by c in L*, 
index the finite number of integral orbits on = Sjm2{W) with characteristic polynomial f{x). 

We summarize what we have established for the representations y = A2(W^)andy = Sym2(W^). 

Proposition 9 Let V denote either the representation f\2{W) or Sym2(lV) ofG. Let f{x) be a poly- 
nomial of degree 2n + 1 with coefficients in Z and non-zero discriminant in Q; ifV = A2iW) we 
further assume that f[x) = xg{x'^) for an integral polynomial g. Then the integral orbits ofG{Z) on 
V{7j) with characteristic polynomial f{x) are in bijection with the equivalence classes of pairs (/, a) 
for the order i? = Z[x]/(/(a;)) defined above, with the property that the bilinear form { , ) on I {given 
by (9) or (10), respectively) is split. 

In terms of Proposition 4, the integral orbit corresponding to the pair (/, a) maps to the rational 
orbit of SO(iy)(Q) on V{Q) corresponding to the class of a = a^^. Here we view a as an element of 
{K*/NE*) when V = A2(1V), so L = E + Q and = K + Q. When V = Sym2(Vr), we view a as 
an element of {L* /L*'^)n=i. 

Finally, we remark that it would be interesting and useful to develop a theory of cohomology 
that allows one to describe orbits over the integers as we have in the cases above. For example, let us 
consider again the representation V of the group G = PGL2 over Q given by conjugation on the 2x2 
matrices v of trace zero. Then this is the adjoint representation, and is also the standard representation 
of SO3 = PGL2. The ring of invariant polynomials on V is generated by q{v) := — det(t'), and the 
stabilizer G^ of a vector with q(v) = d 7^ is isomorphic to the one-dimensional torus over Q which 
is split hy K = Q{\/d), and all vectors w with q{w) = q{y) 7^ lie in the same G(Q)-orbit. 

A natural integral model of this representation is given by the action of the Z-group G = PGL2 
on the finite free Z-module of binary quadratic forms ax"^ + bxy + cy'^. This is equivalent to the 
representation by conjugation on the matrices of trace zero in the subring Z + 271 of the ring TZ of 
2x2 integral matrices. In this model, the invariant polynomial is just the discriminant d = b"^ — Aac of 
the binary form. The content e = gcd(a, 6, c) is also an invariant of a non-zero integral orbit. 

We may calculate the G(Z) -orbits on the set S of forms with discriminant d E Z — {0} and 
content e = 1 (so the binary quadratic form is primitive) via cohomology. Let O = 0{d) be the 
quadratic order of discriminant d. Then the stabilizer G^, of such an orbit in PGL2 is a smooth group 
scheme over Z which lies in an exact sequence (in the etale topology) 

1 ^ Gm ^ Reso/z Gi, 1. 

Furthermore, the Z-points of the quotient scheme G/G^ can be identified with the set S. Hence the 
orbits in question are in bijection with the kernel of the map 7 : H^(Z, G^) — )■ H^(Z, PGL2) in etale 
cohomology. Since if^(Z, PGL2) = 1, the orbits are in bijection with the elements of H^{Z,Gv). 
Since H^{Z, Gm) = H'^{Z, Gm) = 1 the long exact sequence in cohomology gives 

H\Z, G,) = H\Z,Reso/zGrn) = Pic(O). 

Hence the orbits of PGL2(Z) on the set S of binary quadratic forms of discriminant d ^ and con- 
tent 1 form a principal homogeneous space for the finite group Fic{0{d)) of isomorphism classes of 
projective 0((i) -modules of rank one. Thus the number of primitive integral orbits contained in the 
rational orbit of discriminant d is given by the class number of 0(d). 
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